Expose DCR config in operator CRD for OAuth2 upstreams

Closes #5040

Summary

Phase 2 of the DCR story (#4978) finishes by surfacing Dynamic Client Registration (RFC 7591) in the operator API so Kubernetes users can configure DCR on VirtualMCPServer OAuth2 upstreams. The CRD changes were intentionally held back until the authserver plumbing landed, and this PR threads the new dcrConfig field from the CRD through conversion to the existing authserver.DCRUpstreamConfig runtime type. Validation lives in CEL on the CRD plus a defense-in-depth check at reconcile time, and the initial access token follows the same Secret-ref → env-var pattern already used by ClientSecretRef.

Changes Made

Operator API (cmd/thv-operator/api/v1beta1/)

• Add DCRUpstreamConfig type with discoveryUrl, registrationEndpoint, initialAccessTokenRef (a corev1.SecretKeySelector), softwareId, and softwareStatement fields.
• Add optional dcrConfig *DCRUpstreamConfig to OAuth2UpstreamConfig and make clientId optional.
• CEL validation enforces: exactly one of clientId or dcrConfig on OAuth2UpstreamConfig; exactly one of discoveryUrl or registrationEndpoint inside dcrConfig; and a fail-closed discriminator on UpstreamProviderConfig so unknown future provider types are rejected at admission instead of silently accepted.
• MaxLength=255 on softwareId and MaxLength=4096 on softwareStatement to bound CR size against etcd object limits.

Conversion (cmd/thv-operator/pkg/controllerutil/authserver.go)

• Map DCRConfig from the CRD onto authserver.DCRUpstreamConfig in the OAuth2 upstream branch, including resolving InitialAccessTokenRef to a TOOLHIVE_UPSTREAM_DCR_INITIAL_ACCESS_TOKEN_<PROVIDER> env var sourced from the referenced Secret — same pattern as ClientSecretRef.
• Call the exported ValidateOAuth2DCRConfig from BuildAuthServerRunConfig so malformed ClientID/DCRConfig pairs that bypass admission still fail at reconcile time rather than at authserver startup. OIDC and OAuth2 branches were split into helpers to keep gocyclo within limits.

Authserver runtime (pkg/authserver/, pkg/oauthproto/, pkg/auth/)

• DCR credential store and resolver: pkg/authserver/runner/dcr.go, dcr_store.go. In-memory store keyed by (Issuer, RedirectURI, ScopesHash), no TTL (RFC 7591 registrations are long-lived). The key shape is designed to compose a Redis segment in Phase 3 without redefining the canonical form.
• EmbeddedAuthServer now owns a DCRCredentialStore and resolves DCR for any OAuth2 upstream with DCRConfig before building the upstream config. The resolved ClientSecret is overlaid on the built upstream.OAuth2Config via a new applyResolutionToOAuth2Config helper.
• Each UpstreamRunConfig element is shallow-copied and its OAuth2 sub-config deep-copied before DCR resolution to honor the "Copy Before Mutating Caller Input" rule.
• The /oauth/register handler emits a structured Info log on success with issuer, client_id, software_id, token_endpoint_auth_method, and scopes. software_id is threaded through ValidateDCRRequest and capped at 256 printable-ASCII characters.
• MonitoredTokenSource gains required upstream and clientID constructor parameters so DCR remediation warnings carry correlation fields, and the runner prefers the DCR-cached client_id over the statically configured one.
• pkg/oauth is renamed to pkg/oauthproto and the DCR/discovery primitives from pkg/auth/oauth/dynamic_registration.go and pkg/auth/discovery/ are consolidated there. Callers updated.

Generated artifacts

• zz_generated.deepcopy.go, deploy/charts/operator-crds/**/*.yaml, docs/operator/crd-api.md, and docs/server/{docs.go,swagger.json,swagger.yaml} regenerated via task gen / task crdref-gen / task operator-manifests. No hand edits.

Implementation Details

• Fail-closed CEL discriminator on UpstreamProviderConfig. Restructured the previous binary check into a chain ending in explicit false, with a contributor-facing doc comment reminding future authors to extend both the CEL rule and validateUpstreamProvider when adding a new UpstreamProviderType.
• No Unicode smart quotes in kubebuilder markers. gofmt's doc-comment formatter was reintroducing U+201D into != '' markers on every lint-fix. Switched to CEL's size(...) > 0 idiom so regeneration is idempotent.
• Single error log at the boundary. The DCR resolver now wraps step failures in DCRStepError and the boundary caller (buildUpstreamConfigs) emits a single slog.Error via LogDCRStepError, instead of every error branch logging on its own.
• URL-trailing punctuation preserved. sanitizeErrorForLog's URL-stripping regex was greedily consuming sentence-ending punctuation. Trailing terminal punctuation is now split off the match before parsing and re-appended after the query is stripped, with regression coverage for commas, periods, parens, and quoted URLs.

Testing

• cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types_test.go: table-driven CEL validation covering dcrConfig+clientId conflict, both endpoints set, neither endpoint set, valid single-endpoint cases, OIDC-typed provider rejecting an OAuth2 dcrConfig, and the non-DCR clientId-only path.
• cmd/thv-operator/pkg/controllerutil/authserver_test.go: new TestBuildAuthServerRunConfig_InvalidDCR suite covering all four invalid DCRConfig/ClientID pairings at the conversion layer; conversion tests for discoveryUrl/registrationEndpoint paths and initialAccessTokenRef env-var wiring; assertion that the upstream name appears exactly once in scoped error strings.
• pkg/authserver/runner/dcr_test.go, dcr_store_test.go, embeddedauthserver_test.go: full DCR boot path against a mock authorization server, cache-hit short-circuit asserting zero additional HTTP requests, caller's original RunConfig.Upstreams slice element unchanged across calls, TestNewEmbeddedAuthServer_DCRBoot driving the full constructor and asserting dcrStore is populated after boot.
• pkg/oauthproto/dcr_test.go, discovery_test.go: relocated and extended coverage for DCR primitives and AS metadata discovery in the consolidated package.
• pkg/authserver/server/handlers/handler_chain_test.go: TestHandler_issuer so a real wiring bug fails loudly instead of logging issuer="".
• task lint, task lint-fix, task license-check, and the affected unit tests all pass; CRD/deepcopy regeneration is idempotent.

API Compatibility

• This PR does not break the v1beta1 API. dcrConfig is a new optional field, clientId becomes optional but the existing CEL rule requires exactly one of clientId or dcrConfig, so previously-valid CRs that only set clientId continue to validate.

Large PR Justification

The diff is ~2k LoC, split roughly in half between hand-written code and autogenerated artifacts. A clean split would leave the feature in an unreviewable intermediate state.

• Generated artifacts (~1090 LoC, 5 files). Adding DCRUpstreamConfig and reshaping OAuth2UpstreamConfig regenerates four CRD YAMLs (the MCPExternalAuthConfig and VirtualMCPServer CRDs each appear in deploy/charts/operator-crds/files/crds/ and the templates/ mirror), zz_generated.deepcopy.go, and docs/operator/crd-api.md. These are produced by task operator-manifests / task crdref-gen / task gen and are not hand-edited.
• Tightly coupled CRD + conversion + tests slice (~1080 LoC, 4 files). mcpexternalauthconfig_types.go adds the API surface (DCRUpstreamConfig, optional clientId, CEL XOR rules, the unified discriminator check, length caps, the ValidateOAuth2DCRConfig reconcile-time backstop). pkg/controllerutil/authserver.go does the conversion onto the existing authserver.DCRUpstreamConfig runtime type, including the TOOLHIVE_UPSTREAM_DCR_INITIAL_ACCESS_TOKEN_<PROVIDER> env-var binding. The matching test files exercise both layers. Splitting would either ship a CRD that admits DCR but is dropped at conversion time, or conversion code with no API to invoke it — the user-visible feature is the union of the two.
• Review-feedback hardening already factored into separate commits. The last three commits (Tighten and align upstream OAuth2/DCR validation, Refactor ValidateOAuth2DCRConfig and OAuth2 conversion helper, Strengthen DCR env-var assertions and consolidate redundant tests) split the response to the 16-finding consensus review along clear themes — validation tightening, validator API refactor, test consolidation — so each is reviewable on its own diff on top of the original feature commit.

Does this introduce a user-facing change?

Yes. Cluster admins can now configure RFC 7591 Dynamic Client Registration on VirtualMCPServer OAuth2 upstreams via the new dcrConfig block on OAuth2UpstreamConfig, including a Secret-backed initial access token, a softwareId/softwareStatement pair, and either an RFC 8414 discoveryUrl or an explicit registrationEndpoint. ToolHive registers a client with the upstream on first use and caches the resulting credentials in memory.

Special notes for reviewers

• This unblocks Persistent DCRCredentialStore backends (memory + Redis) #4979 (Phase 3 persistent backends). The DCRCredentialStore interface is the drop-in point — no further interface churn is expected when the Redis-backed implementation lands.
• The pkg/oauth → pkg/oauthproto rename is in this PR because the DCR resolver imports the consolidated package; it touches a number of import statements but is otherwise a pure rename with no behavior change.
• MonitoredTokenSource's upstream/clientID constructor parameters are now required (replacing the previous SetUpstreamContext setter). Call sites were updated; the change forces compile-time visibility of the correlation fields rather than relying on a post-construction setter.